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ABSTRACT • ' 

Over the yearfe many criticisms have been ^offered 
against the multiple choice test format. Ambiguous, and emphasizing' 
isolated .information, they are alsothe most difficult objective 0 
tests* to construct. Over-interpretaftion' is a danger of multiple' 
choice examinafibns with students picking- subtle answers the test' * 
makers consider incorrect. Yet, the multiple choice test is a tool 
Offering versatility in measuring educational objectives. It 
meajsurters 'the student's discriminatory thinking, comprehension,, 

and evaluation. Guessing is minimized and the 
on the exam, the .more * representative are the 
sample questions. Perceived ambiguity could allow the scorer to see 
that the student understood the concept being tested 'by explaining 
particular qualifications they perceive and their individual 
resolut:" 
was 

coul<J explain alternate answers* This document deals with 
development and implementation. (Author/CE) 



application , synthesis, 
greater*number o£ i£ems? 



>lutions of the ambiguity. The multiple choice-essay exam* (MCE) 
developed incorporating a wide, right-hand margin s'o the studen't 
14 explain alternate answers* This document deals with its * 
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MiltoYi and Ecfgerl^ (1976) , N McGui^( 1968), Green (1975), Black (1963), 
and Macintosh (1974) summarize the many criticisms of the multiple choice test 
^.forrtiat. Students also offer complaints tjiat multiple choicer test items t'&nd 
,to be ambiguous artdjoften emphasize the trivial. - 

'%McGuire (1968) reports that over half of the multiple choice items he- 

* t \ ' 

studied stressed recall and recognition of isolated information^ Green" 
(1975) points out that multiple ,choice tests are the most difficult objective 
*tpsts to construct. For example, it is often difficult to construct a set, of , 

alternatives to 'a question with only one alternative as'clfearly justifiable; * 

*■ * » > ' 

and it is relatively easy to omit one or more of the qual ifications needed \ 

for correct answer selection: In- addition* there is a tendency for test * 

* . . • ^ " *; 

makers to .include nonfunctional words which interfere with student compre- im 

hens i on. ' Difficulty level is adso'hard to control, and Black (1963) reports 



that afe a result multiple choice tests might be detrimental to brighter 

m . 

.students* According to Black, multiple choice tests are apt to contain 
questions whicfo might confuse brtghte^students wh/3h pick subtle answer^ which 
ttfe^t&st maker considers incorrect: Millman and Pauk (1969) also warn of the 
danger* of overinterpr£ting mul tiple*choice questions. - , 

Yet, multiple choice tests are quite common in college settings and 
Represent the most popular objective^ testing technique (Nunnally, 1972).'' 

A * " ' ' * 

Greefi (1975)' maintains that the multiple. choice test is the best type of 
objective test for measuring a Variety of educational 'objectives. Multiple 
choice tests are .versatile and require discriminatory thinking on th-e part 
of the students Ttfe test format is useful for measuring knowledge, , * 



comprehension, application, synthesis, ,and evaluation. It minimizes guessing* 

. ^found in true-false -tests and is adaptable to numerous fields. 

• Multiple choice tests htfve several other inherent advantages. The greater 

the number.of items on an exam* the mora representative are the sampled questions 

Multiple choice tests are also more objective than other formats, e.g., essay 

exams,, because two or mofe scorers will arrive at the same score after a key 

is prepared. According to Milton and Edgerly (fi976)«, however, unless the 

test is constructed carefully, much of the score could be arbitrary. ' 

Attempts have been made to mddify the multiple choice format so that its . t 

"strengths could be retained while^alleviating several of its major criticisms. 

The most frequently cited modification is the partial knowledge or 

* confidence-testing procedures, (llmsden, 1977, Echternacht, 1972, Curlette, 

1978, Coombs, et. al., 1956, Stanley and Wang, J970, Diamond, 1975) One - 

variation (Jacobs, 1971) has the student assign personal probabilities of , 

correctness to each alternative, and the -student's personal probability of 

the actual qorrect alternative becomes his/her score for the question. Another 

^-L- (Coombs, et.. al . , 1956) scores questionSf py having the students 

eliminate^ the alternatives they decide are incorrect. The more alternatives 

appropriately marked incorrect, the more' credit the student earns. 

Although confidence-testing allows- credit to be assigned for 'partial 

knowledge, it is not without its criticisms. Curlette' (1978) found that r 

there is a method of responding that will increase the student's score * t „ 
to* • 

irrespective of their knowledge. On a practical level, confidence testfng 
formats seem complicated and time consuming £<f* score. A monobasic criticism, >, 
hoWever, is that the confidence-testing option does not sqyarely address the - * 
issuessof 'ambiguity, overintenpretation, and confusion. .The lfterature 




review revealed it is'cfifficult and time consuming for an instructor to create 
_a test which is perceived-to be error fre^ and'not subject to multiple 

interpretations which may have inadvertantly been given in lecture.- 

* * 

* , » » * 

One strategy for correcting {tie .destruction of perceived ambiguity, 

confusion, arrd lack of all necessary qualifications would Be to* allow students 

an alternative method of expressing themselves* on. those items.- This mode . 

1 wouldyserve to indicate to the "scorer that the' students understood ^the 

• x 

concept being tested by explaining the particular qualifications they 'perceive 
and their individual resolutions of the ambiguity and confusion^ Argent " 
(1974) found *that students-are good judges of ambiguity and can easily/point 
out why an incorrect alternative colild be- an appropriate response. This 
strategy* led to the development of a multiple choice exam with a short essay* 
pn+ion which came to be kr/wn as the multiple choice-reSsa/ exam (hereafter v# 
referred to as "MCE"). Structurally, the test booklet appears .to be. a 

4 • , 

."vi.itiple choice test. The only apparent difference is an extremely wide 

right-hand margin, 3 1/2" ttf 4" wide.* (See Attachment 1) The answer sheet 

is d' standard machine scored form, in tyiis case Scantron, #882. 

fhe student was instructed that the test items could be. 'answered in tv/o 

M 

ways. Each question could be answered as if it was a multiple choice question. 

If, however, the student found that the question contained any of the following 

s . < * 

problems—ambigui ty, the appearance of no*right answers, the appearance of 

more than one right answer, different right answers depending upon how the 

question is B interpreted—the student was instructed to place on the machine 

scored answer sheet the answer he or she felt was the one the instructor was 

most likely to consider correct. * Then the stqdent was to write in ^e wide 



•s 



-4- 



margin next to the question a short e§safy explaining the "answer they believed 
to b.e correct, and why. 1 

0 * 

v When the machine scored forms vfere returned to. the -instructor, the short- 
answer essays were reviewed. If the studept had gotten the questvaa correct, 

' . } ' 

the short-answer essay was. not read. If t,he Student had gotten the Question 
wrong, the ho rt- answer essay was carefully read and from zero topfull, credit / 
was assigned. 

* I * . m 

Inl assessing 'the effectiveness of this test format, several outcomes 

were considered important. c /~~ ' ' • 

s ' / 

g Did the students find this format, to be helpful?> 
2., How did the MCE format compare with the .a&am format 'that students usually 
prefer? , ' * \ ' , ' \ < 

3. Did this option significantly raise scores on examinat'vpns? 
4: How often |y|re essays written and how much' time was necessary to 
score them? • ^ . ^ • ' 

5. Was there any relationship between the number of essays written and 
i'lcTjt tnfficuUy? 



Method — : • . • • 

■ — ' » «• 

. ' At the beginning of the class meeting following ^th^e^am, an anonymous 
questionnaire (see Attachment 2) was administered to 118 students in 3 classes. 
The questionnaire included foreed-thoice items regarding the perceived difficulty 
of the MCE exam, its compairson with students 1 preferred exam formats, as well as 
Vts compairson with the standard rnuljh pi e, .choice format. Other questions 
surveyed sfadenffs 1 preferred test format, reasons for their preference, and the * ( 

']■•'*'' 

• .-. ■ • 1 ■. ■■ * • • • 



most frequently encountered te^t format, finally, students^were asked how many 

essays they ha N d written and whether or>not the MCE format was helpful if they had 

or* had not made use of the essay option. * * v * 

Vseven hundred seventy-nine-ex<ams were assessed to determine the number of 

essays per exajn and credit given. . An item analysis of 110 MCE exams Was 

correlated with tfje number of essays wrjtten on fcach test item. MCE exams from 

twel ve' classes (N = 493), were analyzed 'to determine the c>ass average before 

essays were* reviewed, the] class average after the effects of the essays were 

included, and the diffference/between the two averages. The total number 'of 

wt-zys written was categorized by credit given for-- each essay (0, 1/4, 1/2, 3/4,, 
' * * 

1, or already correct). Finally, *he time* to review the "essay portion of the 
exam */as calculated for seven classes in which the MCE exam was given. 

Results—Questionnaire Analysis - — 

. " Table 1 summarizes the students 1 preferred exanrformatsyas well as the 

% ■ * ' , r . * * 

*"«rytz ^the^u most frequently encounter. In responding to the question , concern— 

ing- preferred test formats, a large percentage of the students, 62.62, preferred 

exams with a\multiple, choice format. 'Open-ended responses revealed that the 

largest group of these students felt the multiple-choice test was /the 

easiest for them to t&Ke. According to many of these students, recognition 

was easier than recall; and they knew the right answer was anibng the 

♦ 

alternatives. , 

A second large group reported that their preference was due to their ^ 

i 4 

past success^with this format. They did not amplify. Other, reasons cited 
included a T better opportunity to"gu£ss, less need for memorization, and a 
feeling tfoat multiple-choice questions -helped refresh their memories. Several 

• .. v- ■ . - •. - *- * 



students were concerned with poor grammar and spelling ski f lls r and the multiple 
choice test dtd not- require the demonstration of grammar skills. 

The next most preferred exam format was short essays, which* 16. 5% preferred 
The most frequently cited reason for pre;Pkrring t the short-answer essay* format 
involved criticism of the multiple-choice format. These students believed 
that multiple-choice questions were easy to misinterprets and they found it 
often difficult to discriminafe between alternatives, Short-answ'er essays** 
were straightforward and allowed the students to more fully explain' themselves 
in their own worcft. Several students felt this format was the best test of 
'Knowledge. An additional smctll number of students appreciated the .opportunity 
to^express their own views and^&piniOns. • 

Only 2.6% of the total sample .selected long essays as their- preferred 
tpst forpiafc. p 

In response to the item concerning the most frequently encountered test 
format, 58.0% reported they encountered the multiple choice format most 
frequently. Combinations of exam formats accounted for 1 3 • 4% and short es'says 
wete most' frequently encountered by 11.8% of the students. Over 15% reported^ 
their most frequently encountered test was other than the alternatives listed' t 
on the questionnaire Inspection pf the raw data revealed that the majority 
of those^students had exams with a problem aolving format typical of engineer- 
ing and math curricula. 
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Of the students who^ffrpferred .multiple choice tests » 62.2%* actually 
encountered multiple choice tests most frequently. 1 Only 15% of the ^tudents 
who preferred the short essay exams encountered short essay exams most - 
frequently. The most frequently ^countered exam for those who preferred * 
short essays was the multiple choice format. \ * 



r 



In rating the. MCE format v aqa ins t the standard multiple choice v format, 84.5% 
rated the*MCE much better or somewhat falter* (See Table 2) Only 5.2% 
rated it somewhat worse'than the standard multiple choice, format. Of those' 
who preferred short-answer formats, 80% found the MCE fSrm&t preferable to 
most multiple choice formats. * - N 

In rating the MCE wi^th their preferred test, format, -68. 7% rated it much ■ 
better or somevyhat better than their preferred" format. Only 9.6% rated the 
MCE worse or somewhat worse than their preferred test* format. Of* those who 
preferred multiple' choice, formats, 77.8$ rated the MCE much better than 
iiiultrple choice exams. Of the students preferring short-answer formats, r 

V 4 ..... . t 

60% found 'the MGE format more helpful". Only^those students who preferred 
long essays rated the MCE more- consistently be.low their preferred format.* 

' , ■ i* \ * < ' 

At least one essay was reported written by 41.2%"of the students. Of • 

the students writing' essays, 98% felt the option of writing, essays made the 

* $ 

» ' ?- , * y % * 

tp$f. .easier for them, jhe majorityof the open ended* responses to this 

question indicated .{hat the students 'appreciated the opportunity to explain • 

or clarify tfreir answers. Some Ttudents pointed to clarifjgng confusing 

questions, while other students clarified their answers because they themselves 

were" confused. "Several students stated that they appreciated being treated 

^ * - 

as /individuals with this format. They reported they .felt they could have 

S * * r 

different perceptions than the instructor ,apd also be correct. 1 Another small 

number V)f students reported that the option helped them organize their thtfughts 

'andenabled them to select the correct answer. A few students stated 

specifically that it reduced anxiety for them. * 

Students reporting writing no es.says made up 58.8% of the sample. . 

However, 75.4% of these students felt the* option was stiVTvhel pful . .Their 
• ' % - v • - . v \ - 



open ended responses fell into three overlapping categories plus a residual. 

* i ^ ^ • 

The most frequently, cited reason seemed to center around a reduction of 

anxiety or tension. This ^reduction occurred while studying for thfe test, 

and/or while takijig the exam itself. The second most frequently cited reason 



v^as an appreciation that if they had encountered^! ambiguous question they 

had the' opportunity to explain thefr answers. Th£ third' category focused or 

* 

having' been given the choice to- use the option >if they wished, + 



v 



Test Analysis 



Seven hfundred seventy-nine tests were analyzed; and of the 779 .students 
taking the exam, 425 or 54.5% wrote a* total o'f 1791 essays. Of the essays 
written, ,64.7% were already scored as correct on the machine-scored answer 
sheet and were therefore not read. A total of 19*8% of the esSays received 6 
partial credit (.25 to^75 points) and $-.6% received full' credit. No credit" 



i 



was given to 9.9% of the essays. - r A 
' . ' ■ ' V 

The essay option had-\tery lHtle impact on the final average of the 

tests. The' average percent change was only 0.5%, and only an average of 

lr25 students per test experienced a gain which led to a raise in the letter 

grade for the exam. (See Table 3) The relationship between th^ item * 

analysis and number of. essays written produced a modest -correlation of 

r = .35, p < .003. . . - . * ■ 

The *exams required 'an average'of 23.1 minutes to score.' * This is an** 

o veres timajte- sinte it fl includes time which was used to categorize responses " 

for research purposes. Another indication of the investment of time required 

was the average number of short-answer essays 'that had; to .-be read per test. . 



• . ■ i > • . - ' . . • : . 

x ft The average number of essays^ per* class per 'test was 64. Since an average of 
over 41" of those £4 had already been scored as correct on the mach'ioe scored . 
answer sheet, an average, of only 23 questions per class ('9las9 size < 
^ . ' approximately 40) required re.ad'ing'. 

* : Discussion , ' ' . \ 

" v The resultsfcof the questionnaire indicate that students believe tfcia^the 
MCE option of writing short answers to. ambiguous or confusing multiple choice 
questions was definitely helpful. Even students wl^o wrote no essays fo\md v 
the option anxiety reducing. Students generally preferred the MCE format . 

ovej* the standard multiple choice 'format ; .and in a surpH'sing 68.7% of the 

* / *» * 
t sample, students rate the MCE somewhat or mucfr better than their preferred 

test format. Although the evaluation of the MCE test was only conducted 

immediately^after the examination and before the students received their 

grades, Attkisson (1975) found that the student evaluation of an examination 

immediately after the testing period and an evaluation conducted one week 

^ later (following a review of the exam, Receipt of scores, -and assignment of * 

grades) were the same. , In addition these results of the current study arp not 

Jikely to be. explained as a simple version of the Hawthorne effect because in subs£- 

quent quartej/s student evaluations, have remained high even when the essay option ' 

* - was built intcthe course structure "and «given very little attention 

-One reaspn the MCE format may ^be perceived as being so helpful is that 

* it combines major*elements of the two most preferred exams, multiple choice 

* . % ' ' # 1 

and short-answer essay.- In fact, the MCE removed many 'of the criticisms of r 

^ * multiple choice tests*" offered by tho^e who preferred short-answer essays. 

-The MCE .appears to be- superior t'ceach of them individually since a 'large 

'* - ^ 

percentage of those -students who preferred short answer. on multiple choice 

o - ' 
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rated the . MCE as better thari their prefeJ^ed format, 6d.02,arrtJ 77-. 8 m . * x ► 
respectively. ' " ' • 

Another possible reason for the success' of this format from the students 

* * ■ * « 

perspective was articulated by several students \$ho believed that this option^ 

gave them more control over the situation since' their -possible idi/osyncratic 

interpretations, would be considered. This option was therefore ahxiety 

reducing regardless of whether or hot a student used it. ^TJis iY t#pretation 

is mor^onsistent with* a higher j^rder version of the Hawthrorie effect which 

suggests tha p t one component of increased productivity is a sense/of control in 

work group. • ' ' ' f 

U is interesting to note that\al though -the ^udents- feel „ the MCE test 

format is helpful it does not 'significantly affect their scorers. ;lt only 

. rai v ed*the,'class-average .22 of a point, ancFonty an average of J, 25 students 
J • A " r • « 

received gains that -raised thei r letter grades (for the exam. The MCE, while 

making-an insignificant 'impact on the class average, apparently has a IJheat 

impaction the individual/ student'^ peremption of the £xam and its fairness. 

v * Most of the essays were wrltten^ott questions th^at;. were already correct" - 

on the machine scored answer sheet? This finding is consistent with . research 

that demonstrates that people, can ma.ke accurate responses or discriminations 

although they have very ,}ittle or no confidence jn their, accuracy .(Adatos, 1957) 

It is possible that clarifying these answers may haye had the effect of reduc- 

ing anxiety in general and as a result may have raised the class average in 

general/ Siriee.it was not feasible irt thl Study to'have a xdrttno 1 group 

which took' a' standard mul tipje. choice exam without the MCE option, no comment 

can be made. , . 



Further research will focus on c<Dntrol group comparisons, an indepth 
analysis of individual styles of responding, and the criteria by which partial 
credit is given. ... 

It is pbvious, however, that the MCE format is a marked improvement, over 
the standard multiple choice format. The exam is as ea^sy to administer *s 
the standard multiple choice test find requires only a modest investment ^ * 
additional time by the instructor. As\universi ty budgets become increasingly 
"^trained and class sizes increase, instructors who prefer essay type e*ams 
1 wi'M not be able to use that format. Th*e MCE format appears to be, a viable 
alternative that is perceived to be fair and equitable by the students. ^ 
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Preferred Exams 



Most frequently t , % 
Encountered Exams* 



Format ' 

4 - * 


•N • 




N 

♦ 


■ % 


t 

.Multiple', Choice , 


72 


62.6% 


» 

69 .' * 


.58.0% 


Short Essay 




16.5 


. 14 V 


11.8 


• Long Essay 


3 


2.6 


\ ; 2. * 


** 1.7 


' *■ True/ False % 


9 


« 7.8 • 


* 0 ' 


0.Q, 


Matching 


8 


7.0 


. 0 ^ 


0.0 


..v/Oral - 


4 


' ' 3.5 


. 0 


' % 0 .0 


« Combinations 


0 


0.0 


' 16 " 


'13.4 • 


Other 1 




-0.0 


18 


■15.1 




' 1)5 


700.0 , 


115 


100. 0 - 


* • 
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Table 2 



MCE Compared 
to 'Standard " 
Multiple Choice 



N 



% 



im MCE Compared 
to Preferred 
Test, Format 

N 



■"uJi Better 


' 72 


62.1 


41 


35.7 


So^ehwat Better 


•.: 26 


22.4 


38 


33.0 


AoOwi: "the \ame 




10.3 


25- * 


21.7 


Somewhat Worse 


" ' 6 


5.2 


. 7 


' 6.1 


Much'-Horse - 


/ 

• • < 0 


Q.O 


4 


.3.5 




116 


100.0 


' 115 


100.0 



V 
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Table 3 



Numbervpf 
Students 


• 

Number of 
Essays 
Wri tten 

III 1 l« l# V» * ' 


• Class 
Average 
Without 
Essay, 

0 


Class 
Average 
With 
Essay 


4 

Change 


Percent 
Change 


41 


13 


. 37">22 


37.24 


> .02 pts. 




31 


40 


39.58 


39.89 


•31 


. A 


41 


52 


39.95 


40." 1-2 


" .17 • • 


v4* 


32 


19 


41 . 22 


41 . 04 


. 1 L 




33 


17 


40.06 


" 40.15 


v .09 • 


.2 


36 


35 


• 39.50 


39.66 


.16 • - 


.4 


27 


93 


41.37 


42.05 


.68 , . 


1.6, 


43 


• 81 


• 3«.77 


35.08 


.31 * " 


-.8 * 


.35 


67 . 


• 35.86 


36.24 r 


'• .38 


M.l 


-60 


, 46 '• 


36.57 


36.78 


' • 21 


.6 


59 


44- 


40.57 


40.71 


• .14 

* 


.3 


55 


49 . 


41.42 


40.52 

c 

ave. 


.10 

= .22 


.2 
.5%. 



Number of 
Letter 
Grades 
Changed 



0 
1 
2 
1 
1 
1 
2 

.3' 
0 
2 
0 
2 



1.25 
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11. In the film on aciiievfent f high achievers . a s ch jj cyt 
- had parents who':' * 

a. stressed their child 1 s independence. 

b. always underestimated their child's potential. . 

c. verbally "hassled" their child to higher and , 
higher levels of performance. 

d. were fairly representative .of the general 
population. -* 

12. High "nee^jchievement" individuals: 

a. prefer tasks of intermediate difficulty. 

b. prefer. tasks in which there is concrete feedback. 

c. may be' an important determinant for the economic 
growth of entire societies. 

• d. all of the above, 
v 

13. Cross-cultural achievement* motivation training on 
Indian businessmen: 

* 

a. was largely unsuccessful since achievement 
motivation appears to be inborn. 

b. wa^largely unsuc c e ssfu l becau se achievement 
motivation seems unique to" our culture. 1 

c. wlas successful in terms of increased business 
sorts, capital investment-/ and earnings. 

d. Ifed to. the unprovoked attack on Fort Apache, 
Arizona. . 

4. One probleift with need ^achievement training is that: 

a. ft does not increase motivation. 

b. incerased motivation may lead to an unwanted push, 
for organizational change. 

c. it is very difficult to-generate on int&nal , 
standard of excellence.' 

d. * all of the above. 

>. Operant conditioning theory suggests that in order to ■ 
"motivate 11 employees: • 

a. make sure that positive consequences follow 'the 

behavior you want them to perform. T Jk 
bl -ycli must first uncover the persorTs inner mbtives. 

c. make sure th§ person pnly* takes moderate risks; • - * 

d. ' provide^ an extensive physical education program-. 

. Which of the following* is an example of* referent 

power in leadership? • * ^ ' 

? • 
tt reward and punish yout subordinates ; 
ic personality of Ronald Reagen * 

c. expertise in the task at hand 

d, the rights of Congress to enact laws 



'MULtrf PLE- CHO I CE ESSAY" TEST ANALYSI S 



Compared to exams- fo^ other courses tf this exam was: 
' more difficult;/ > * l ' 



about the sam£ r 



easier 



(a) In general, I prefer; (#1 = most preferred, Ji 
, \ m ultiple-choice ' ' . 
l ong es§ays (one or two question tests) 



short essays 



true- false 



matching 



v 



oral exaras *- 



other 



(b) Why do you have this preference? 



Attachment 2 



le xt m est, etc.,) 



(c) Which are the two mos^t frequent 'tests you usually have?' 
1- most frequent » 



2. next f requent_ 



How would you ra'te^the "Multiple Choice- Ess ay"?. 



(a) 



J^jch better than npr generally 'preferred test. 



/Somewhat better than my generally preferred test. ^ 

About the same as ng/ generally preferred test. 

— • j$ * 

/Somewhat worse than my generally preferred tes*t„ 
_Huch less than my generally preferred test-' * y / 

_Much better than most multiple choice test formats. 
_Somewhat better than most multiple choice test formats. 

About tfie same as most multiple choice test formats.' 

Somewhat worse than most multiple choice, test, formats. 



_Much less than most multiple choice test formats* 

. '• . v 20 



(OYER) 



t wrote essays on . questions.*- 

fa) If you wrote essays. or> 0 (none), did this option make the test easier 
for you in any way? * ^ t * ' * . 



(b) If you wrote essays \>n 1 or more questions, how do you feel this option 
helped you? " • 



Any additional comments. 



